Kernel and user level execution trace analysis for multi-core distributed Linux systems

The presentation will briefly introduce the organization of the DORSAL laboratory and the collaborative research and development projects undertaken with the financial support of industrial partners, including Ericsson, and the Canadian and Quebec governments. Then, several recent results in the area of kernel and user level tracing and trace analysis will be outlined, with emphasis on real-time, distributed, multi-core and heterogeneous Linux systems. Different widely used open-source tools developed collaboratively in the DORSAL laboratory, such as LTTng, Trace Compass and the Common Trace Format, will be described as well as how they were exploited to quickly find the critical path and uncover hard to find problems in several real industrial systems

Fast Recompilation of Object Oriented Modules

Once a program file is modified, the recompilation time should be minimized, without sacrificing execution speed or high level object oriented features. The recompilation time is often a problem for the large graphical interactive distributed applications tackled by modern OO languages. A compilation server and fast code generator were developed and integrated with the SRC Modula-3 compiler and Linux ELF dynamic linker. The resulting compilation and recompilation speedups are impressive. The impact of different language features, processor speed, and application size are discussed

Disks, Partitions, Volumes and RAID Performance with the Linux Operating System

Block devices in computer operating systems typically correspond to disks or disk partitions, and are used to store files in a filesystem. Disks are not the only real or virtual device which adhere to the block accessible stream of bytes block device model. Files, remote devices, or even RAM may be used as a virtual disks. This article examines several common combinations of block device layers used as virtual disks in the Linux operating system: disk partitions, loopback files, software RAID, Logical Volume Manager, and Network Block Devices. It measures their relative performance using different filesystems: Ext2, Ext3, ReiserFS, JFS, XFS,NFS

virtFlow: guest independent execution flow analysis across virtualized environments

An agent-less technique to understand virtual machines (VMs) behavior and their changes during the VM life-cycle is essential for many performance analysis and debugging tasks in the cloud environment. Because of privacy and security issues, ease of deployment and execution overhead, the method preferably limits its data collection to the physical host level, without internal access to the VMs. We propose a host-based, precise method to recover execution flow of virtualized environments, regardless of the level of virtualization. Given a VM, the Any-Level VM Detection Algorithm (ADA) and Nested VM State Detection (NSD) Algorithm compute its execution path along with the state of virtual CPUs (vCPUs) from the host kernel trace. The state of vCPUs is displayed in an interactive trace viewer (TraceCompass) for further inspection. Then, a new approach for profiling threads and processes inside the VMs is proposed. Our proposed VM trace analysis algorithms have been open-sourced for further enhancements and to the benefit of other developers. Our new techniques are being evaluated with workloads generated by different benchmarking tools. These approaches are based on host hypervisor tracing, which brings a lower overhead (around 1%) as compared to other approaches

Tracing and profiling machine learning dataflow applications on GPU

In this paper, we propose a profiling and tracing method for dataflow applications with GPU acceleration. Dataflow models can be represented by graphs and are widely used in many domains like signal processing or machine learning. Within the graph, the data flows along the edges, and the nodes correspond to the computing units that process the data. To accelerate the execution, some co-processing units, like GPUs, are often used for computing intensive nodes. The work in this paper aims at providing useful information about the execution of the dataflow graph on the available hardware, in order to understand and possibly improve the performance. The collected traces include low-level information about the CPU, from the Linux Kernel (system calls), as well as mid-level and high-level information respectively about intermediate libraries like CUDA, HIP or HSA, and the dataflow model. This is followed by post-mortem analysis and visualization steps in order to enhance the trace and show useful information to the user. To demonstrate the effectiveness of the method, it was evaluated for TensorFlow, a well-known machine learning library that uses a dataflow computational graph to represent the algorithms. We present a few examples of machine learning applications that can be optimized with the help of the information provided by our proposed method. For example, we reduce the execution time of a face recognition application by a factor of 5X. We suggest a better placement of the computation nodes on the available hardware components for a distributed application. Finally, we also enhance the memory management of an application to speed up the execution

Hypertracing: Tracing through virtualization layers

Cloud computing enables on-demand access to remote computing resources. It provides dynamic scalability and elasticity with a low upfront cost. As the adoption of this computing model is rapidly growing, this increases the system complexity, since virtual machines (VMs) running on multiple virtualization layers become very difficult to monitor without interfering with their performance. In this paper, we present hypertracing, a novel method for tracing VMs by using various paravirtualization techniques, enabling efficient monitoring across virtualization boundaries. Hypertracing is a monitoring infrastructure that facilitates seamless trace sharing among host and guests. Our toolchain can detect latencies and their root causes within VMs, even for boot-up and shutdown sequences, whereas existing tools fail to handle these cases. We propose a new hypervisor optimization, for handling efficient nested paravirtualization, which allows hypertracing to be enabled in any nested environment without triggering VM exit multiplication. This is a significant improvement over current monitoring tools, with their large I/O overhead associated with activating monitoring within each virtualization layer

ros2_tracing: Multipurpose Low-Overhead Framework for Real-Time Tracing of ROS 2

Testing and debugging have become major obstacles for robot software development, because of high system complexity and dynamic environments. Standard, middleware-based data recording does not provide sufficient information on internal computation and performance bottlenecks. Other existing methods also target very specific problems and thus cannot be used for multipurpose analysis. Moreover, they are not suitable for real-time applications. In this paper, we present ros2_tracing, a collection of flexible tracing tools and multipurpose instrumentation for ROS 2. It allows collecting runtime execution information on real-time distributed systems, using the low-overhead LTTng tracer. Tools also integrate tracing into the invaluable ROS 2 orchestration system and other usability tools. A message latency experiment shows that the end-to-end message latency overhead, when enabling all ROS 2 instrumentation, is on average 0.0033 ms, which we believe is suitable for production real-time systems. ROS 2 execution information obtained using ros2_tracing can be combined with trace data from the operating system, enabling a wider range of precise analyses, that help understand an application execution, to find the cause of performance bottlenecks and other issues. The source code is available at: https://github.com/ros2/ros2_tracing.Comment: 8 pages, 8 figures, 3 table

Fine-grained preemption analysis for latency investigation across virtual machines

This paper studies the preemption between programs running in different virtual machines on the same computer. One of the current monitoring methods consist of updating the average steal time through collaboration with the hypervisor. However, the average is insufficient to diagnose abnormal latencies in time-sensitive applications. Moreover, the added latency is not directly visible from the virtual machine point of view. The main challenge is to recover the cause of preemption of a task running in a virtual machine, whether it is a task on the host computer or in another virtual machine. We propose a new method to study thread preemption crossing virtual machines boundaries using kernel tracing. The host computer and each monitored virtual machine are traced simultaneously. We developed an efficient and portable trace synchronization method, which is required to account for time offset and drift that occur within each virtual machine. We then devised an algorithm to recover the root cause of preemption between threads at every level. The algorithm successfully detected interactions between multiple competing threads in distinct virtual machines on a multi-core machine

Mythe et sacré : le pouvoir des mots dans le Seigneur des anneaux

D'enfant intrigué par des inscriptions en gallois, Tolkien évolue vers un apprentissage précoce du latin, du grec et de nombreuses langues anciennes. Sa passion pour les langues ne se dément pas: bientôt il travaille à l'élaboration de langues inventées basées sur le finnois, puis il fonde un club de lecture de sagas en norrois. Dans son étude des langues, il s'intéresse aux anciens textes qui recèlent leur évolution: très vite, pour lui, le langage et le mythe sont indissociables. Le langage génère du mythe et ce dernier survit à travers les langages. Du propre aveu de l'auteur, c'est pour donner vie à ses langues inventés qu'il créa leur monde: « [...] il lui fallait une « histoire » pour lui servir de base. [...] Il mettait ce langage au point, maintenant il lui fallait trouver les gens dont ce serait la langue. » (Carpenter, 2002). En marge de ses recherches universitaires, il construit donc sa mythologie personnelle, un travail étalé sur toute une vie dont quelques vingt ans sur le seul Seigneur des Anneaux pour lequel il s'attarde minutieusement sur chaque mot: « It was begun in 1936, and every part has been written many times. Hardly a word in its 600 000 or more has been unconsidered. » (Carpenter, 1981). En parcourant la biographie de ce linguiste-écrivain, qui peut se surprendre de voir le langage comme personnage principal de son oeuvre Le Seigneur des Anneaux? Son récit se veut une reconstruction moderne basée sur les légendes anciennes qu'il affectionne et il y inclut à cet effet nombre des éléments langagiers, relevant souvent de la pensée mythique, qui donnent à ces récits anciens leur saveur particulière. Parmi ces éléments, nous avons relevé le traitement accordé à la nomination et celui accordé aux manifestations de la parole poétique que sont ces chants et poèmes que l'on retrouve égrenés dans la narration en prose. En premier lieu, nous exposerons l'intérêt de l'auteur pour le langage et les littératures médiévales (principalement germaniques et finnoise), notamment pour leurs techniques langagières particulières. Nous nous appuierons en cela sur ses essais consacrés aux sources et à la genèse du texte (Tolkien, 1981, 1997). Puis, nous mettrons en relief ce qui, de la pensée mythique, éclaire la perspective et l'utilisation par Tolkien du nom et de la parole poétique avec l'appui de l'essai de Mircea Eliade: Aspects du mythe. Ensuite, il nous faudra étudier la littérature médiévale germanique avec l'aide des ouvrages de Régis Boyer y ayant trait. Nous pourrons ensuite nous pencher sur la question du traitement de la nomination dans l'oeuvre de Tolkien: quel est sa nature, son sens et ses fonctions pour le récit et qu'est-ce qui la rattache à la pensée mythique? Enfin, nous nous intéresserons aux nombreux chants et poèmes propres à l'oeuvre. Nous verrons comment ils définissent des cultures, relatent un passé et des mythes; bref, comment ils ajoutent à la cohérence et à la profondeur de l'univers fictif dépeint. Ainsi, et plus que de simples artifices stylistiques, ces manifestations langagières singulières qui imprègnent Le Seigneur des Anneaux participent de la structure même du récit et sont étroitement liées à la pensée mythique: le langage, le mot, a un véritable pouvoir chez Tolkien. Voilà ce que ce travail tendra à démontrer. ______________________________________________________________________________ MOTS-CLÉS DE L’AUTEUR : Mythe, Nomination, Parole poétique, Fantasy